Summarizing image collection using a social network

ABSTRACT

A method for reducing the number of images or the length of a video from a digital image collection using a social network, includes receiving a digital image collection captured by a user to be viewed by a viewer; wherein the viewer and the user are members of the same social network and using a processor to access the social network to determine a relationship between a viewer and the user. The method further includes using the processor to determine a set of summarization parameters based on the relationship between the viewer and the user and using the processor to reduce the number of images or the length of the video from the digital image collection using the determined set of summarization parameters to be viewed by the viewer.

FIELD OF THE INVENTION

The present invention relates to using social relationships from asocial network to summarize image collections that can contain stillimages or videos.

BACKGROUND OF THE INVENTION

In multimedia social networks, there are a flood of images and videosfrom many different members. People generally only look at a smallfraction of the images or videos that their contacts in the socialnetwork publish. In an effort to address the problem, some peoplecondense (summarize) their videos when sharing videos, and share only asmall portion of the images.

There are methods for automatically summarizing a collection of imagesor videos. For example, in U.S. Pat. No. 7,630,562, there is a methodfor automatically summarizing a video (i.e. editing the video to producea shorter version of the video) using a singular value decomposition.U.S. Pat. No. 7,639,275 describes a method of summarizing video contentincluding football by retaining the highlights of the video and omittingnon-highlights. In this manner, a viewer can watch the game in a shorteramount of time than would be necessary to watch the entire game.

U.S. Pat. No. 5,956,026 provides a method identifying a hierarchicalcollection of key frames in a digital video. A video summary includes ofthe collection of key frames and the video can be browsed by viewingjust the key frames.

U.S. Pat. No. 7,483,618 provides a method for generating a video summaryin which content of low quality or very little or no interest isidentified and removed.

SUMMARY OF THE INVENTION

In accordance with the present invention there is a provided a methodfor reducing the number of images or the length of a video from adigital image collection using a social network, comprising:

(a) receiving a digital image collection captured by a user to be viewedby a viewer, wherein the viewer and the user are members of the samesocial network;

(b) using a processor to access the social network to determine arelationship between a viewer and the user;

(c) using the processor to determine a set of summarization parametersbased on the relationship between the viewer and the user; and

(d) using the processor to reduce the number of images or the length ofthe video from the digital image collection using the determined set ofsummarization parameters to be viewed by the viewer.

It is an advantage of the present invention that an effective method isprovided for summarizing images and videos to share between people in asocial network. The summarization of the images and videos is based onthe relationship between the people, and has the effect of permitting aperson to share more content with others in the social network who aremore interested and less content with people who are less interested.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is pictorial of an online social network;

FIG. 2 is a block diagram of an image capture device and processingsystem that can be used to implement the present invention;

FIG. 3 is a block diagram illustrating the social network mediasummarizer 104; and

FIG. 4 shows illustrative results of the present invention for a socialnetwork containing four people.

DETAILED DESCRIPTION OF THE INVENTION

The present invention describes a method for summarizing a collection ofdigital images or summarizing videos for presentation within an onlinesocial network. As used herein, a collection of digital images is a setof digital images, a set of 2D or 3D images, or 2D or 3D videos, i.e. atime series of 2D or 3D images or a series of images in a video alongwith the accompanying audio.

In an online social network, people are connected by links orconnections if they are socially connected with one another. Theconnection between a pair of people can indicate friendship, a coworkerrelationship, an acquaintance relationship, a set of common interests, acommon membership (e.g. both people are members of a club), or an actualfamilial relationship such as spouses, or a mother-daughterrelationship. In some embodiments, a social network is represented as agraph, where vertices (nodes) of the graph represent people, and edgesrepresents a connection between the pair of people. FIG. 1 shows arepresentation of an online social network.

The social network is represented as a collection of data that exists oncomputer-readable memory. The social network includes a set ofconnections 424 or links between users (individuals), and the associatedimages, video, text and other multimedia files associated with eachuser.

FIG. 1 is used to illustrate an online social network. The figureincludes 11 different individuals 420 (A-K) having user accounts. Thereare also digital media assets 430 such as digital images, videos, andaudio clips. The digital media assets 430 are associated with individualuser accounts. The digital media assets 430 contain persons 422 that areidentified with labels. For example, user B has two associated digitalmedia assets 430 (a digital image) and 431 (a video). The first containstwo persons 422 (A and I), and the second (the video) contains oneperson 422 (E). There are also digital media assets 430 that containindividuals that are have not been identified such as unidentifiedindividuals 432, 434, 436, and 438.

Notice that there are three types of connections shown in FIG. 1. First,there are friend connections 424 that directly connect user accounts. Insome embodiments, these connections belong to one or more of severalsub-categories, such as (familial relative, coworker, classmate,friend). Each of these sub-categories can have further sub-categories(e.g. familial relative has sub-categories of father-child,husband-wife, and siblings). A second type of connection is anassociative connection 426 which is a link indicating that a person(e.g. person 420B) has a connection with an indicated person in adigital media asset 430 (such as people A, I and E (indicated as 422A,422I, and 422E respectively) in the case of the person B (indicated as420B)). The fact that the person has associated media assets containingspecific individuals shows an implicit social connection between them. Athird type of connection is an image connection 428 which is a linkindicating that two people appear together in a digital media asset 430,such as the people A and I in a digital media asset 430 that belongs toindividual B. Note that people that appear in digital media assets 430do not necessarily have an individual user account (e.g. person M,indicated as 420M).

The illustration in FIG. 1 can be used to define a connection graph Gover the individuals in the user accounts and identified in the digitalmedia assets.

FIG. 2 is a block diagram of an image capture device and processingsystem that can be used to implement the present invention. A socialnetwork 116 can reside in computer readable memory of a single computer(e.g. a computer server, a personal computer, a laptop, a tablet, acamera, a smart phone, or other device containing integrated circuits),or the social network can reside across two or more computers. Thepresent invention can also be implemented for use with any type ofdigital imaging device, such as a digital still camera, camera phone,personal computer, or digital video cameras, or with any system thatreceives digital images. As such, the invention includes methods andapparatus for both still images and videos. The present inventiondescribes a system that uses an image capture device 30 for capturing animage or video 32. For convenience of reference, it should be understoodthat the image or video 32 refers to both still images and videos orcollections of images. Further, the image or video 32 can be an imagethat is captured with a camera or image capture device 30. Further, theimage or video 32 can be a single-view image (i.e. a 2D image) includinga single perspective image of a scene at a time, or the image can be aset of images (a 3D image or a multi-view image) including two or moreperspective images of a scene that are captured and rendered as a set.When the number of perspective images of a scene is two, the images area stereo pair. Further, the image or video 32 can be a 2D or 3D video,i.e. a time series of 2D or 3D images. The image or video 32 can alsohave an associated audio signal. The system of FIG. 2 contains a display90 for viewing images. The display 90 includes monitors such as LCD,CRT, OLED or plasma monitors, and monitors that project images onto ascreen. The sensor array of the image capture device 30 can have, forexample, 1280 columns×960 rows of pixels. When advisable, the imagecapture device 30 activates a light source 49, such as a flash, forimproved photographic quality in low light conditions.

In some embodiments, the image capture device 30 can also capture andstore video clips. The digital data is stored in a RAM buffer memory 322and subsequently processed by a digital processor 12 controlled by thefirmware stored in firmware memory 328, which can be flash EPROM memory.The digital processor 12 includes a real-time clock 324, which keeps thedate and time even when the system and digital processor 12 are in theirlow power state.

The digital processor 12 operates on or provides various image sizesselected by the user or by the system. Images are typically stored asrendered sRGB image data is then JPEG compressed and stored as a JPEGimage file in the memory. The JPEG image file will typically use thewell-known EXIF (EXchangable Image File Format) image format. Thisformat includes an EXIF application segment that stores particular imagemetadata using various TIFF tags. Separate TIFF tags can be used, forexample, to store the date and time the picture was captured, the lensF/# and other camera settings for the image capture device 30, and tostore image captions. In particular, the ImageDescription tag can beused to store labels. The real-time clock 324 provides a capturedate/time value, which is stored as date/time metadata in each EXIFimage file. Videos are typically compressed with H.264 and encoded asMPEG4.

In some embodiments, the geographic location stored with an imagecaptured by the image capture device 30 by using, for example a GPS unit329. Other methods for determining location can use any of a number ofmethods for determining the location of the image. For example, thegeographic location can be determined from the location of nearby cellphone towers or by receiving communications from the well-known GlobalPositioning Satellites (GPS). The location is preferably stored in unitsof latitude and longitude. Geographic location from the GPS unit 329 isused in some embodiments to regional preferences or behaviors of thedisplay system.

The graphical user interface displayed on the display 90 is controlledby user controls 60. The user controls 60 can include dedicated pushbuttons (e.g. a telephone keypad) to dial a phone number, a control toset the mode, a joystick controller that includes 4-way control (up,down, left, right) and a push-button center “OK” switch, or the like.The user controls 60 are used by a user to indicate user preferences 62or to select the mode of operation or settings for the digital processor12 and image capture device 30.

The display system can in some embodiments access a wireless modem 350and the internet 370 to access images for display. The display system iscontrolled with a general control computer 341. In some embodiments, thesystem accesses a mobile phone network 358 for permitting humancommunication via the system, or for permitting signals to travel to orfrom the display system. An audio codec 340 connected to the digitalprocessor 12 receives an audio signal from a microphone 342 and providesan audio signal to a speaker 344. These components can be used both fortelephone conversations and to record and playback an audio track, alongwith a video sequence or still image. The speaker 344 can also be usedto inform the user of an incoming phone call. This can be done using astandard ring tone stored in firmware 328, or by using a customring-tone downloaded from a mobile phone network 358 and stored in thememory 322. In addition, a vibration device (not shown) can be used toprovide a quiet (e.g. non audible) notification of an incoming phonecall.

The interface between the display system and the general purposecomputer 341 can be a wireless interface, such as the well-knownBluetooth® wireless interface or the well-known 802.11b wirelessinterface. The image or video 32 can be received by the display systemvia an image player 375 such as a DVD player, a network, with a wired orwireless connection, via the mobile phone network 358, or via theinternet 370. It should also be noted that the present invention can beimplemented in a combination of software and hardware and is not limitedto devices that are physically connected or located within the samephysical location. The digital processor 12 is coupled to a wirelessmodem 350, which enables the display system to transmit and receiveinformation via an RF channel 250. The wireless modem 350 communicatesover a radio frequency (e.g. wireless) link with the mobile phonenetwork 358, such as a 3GSM network. The mobile phone network 358 cancommunicate with a photo service provider, which can store images. Theseimages can be accessed via the Internet 370 by other devices, includingthe general purpose computer 341. The mobile phone network 358 alsoconnects to a standard telephone network (not shown) in order to providenormal telephone service.

Referring again to FIG. 2 the digital processor 12 accesses a set ofsensors including a compass 43 (preferably a digital compass), a tiltsensor 45, the GPS unit 329, and an accelerometer 47. Preferably, theaccelerometer 47 detects both linear and rotational accelerations foreach of three orthogonal directions (for a total of 6 dimensions ofinput). This information can be used to improve the quality of theimages using an image processor 70 (by, for example, deconvolution) toproduce an enhanced image 69, or the information from the sensors can bestored as metadata in association with the image and later used by thecollection analyzer 106 of FIG. 3 to determine the quality of each imageor video in the collection. In the preferred embodiment, all of thesesensing devices are present, but in some embodiments, one or more of thesensors is absent.

Further, the image processor 70 is applied to the images or videos 32based on user preferences 62 to produce the enhanced image 69 that isshown on the display 90. The image processor 70 improves the quality ofthe original image or video 32 by, for example, removing the hand tremorfrom a video.

Although FIG. 1 shows an illustrative social network with 11 individualsas members, real-life social networks can contain millions of members.Individual members can have hundreds or thousands of connections in thegraph G. If even a small percentage of connections of a member of thesocial network post a collection of images or videos, the member can bedeluged with a large number of images and videos from friends in thesocial network. For example, if social network member Jerry has 500friends, and of these, 50 post a collection of, on average 50 images andvideos, Jerry will have access to 2500 images and videos. The images andvideos that Jerry really would like to see become difficult to find whenmixed among all the rest of the images and videos from others in thesocial network.

FIG. 3 is a block diagram illustrating a social network media summarizer104 that considers the relationship between pairs of people whendetermining how to summarize the images and videos that are posted tothe social network 116 be a user for viewing by a viewer. In thiscontext, a “user” is a member of the social network 116 who is postingdigital media assets 430 such as images and videos, and a viewer is amember of the social network 116 who is receiving the posted digitalmedia assets 430 for viewing.

A user 100 has an image collection 102 of digital media assets 430 thatcontain images and videos 32. The image collection 102 is processed bythe collection analyzer 106 that produces a collection analysis 108 thatindicates the quality of each image in the image collection 102 andproposes for each video a set of possible summarized (edited andcondensed) versions of the video. The collection analyzer 106 preferablyimplements a method for determining the aesthetic quality of images, asdescribed in commonly-assigned U.S. patent application Ser. No.12/566,706 filed Sep. 25, 2010, entitled “Estimating Aesthetic Qualityof Digital Images” by Cerosaletti et al. The collection analyzer 106determines proposed summarizations for videos in the image collection102. For producing proposals for video summarizations, the collectionanalyzer 106 preferably uses the method described in commonly-assignedU.S. patent application Ser. No. 12/786,480 filed May 25, 2010, entitled“Determining Key Video Snippets Using Selection Criteria”, by A. Deever.This method determines a set of key frames that represent the video.Individual key frames are selected based on several features, includingcamera motion. Video frames having small amounts of camera motion arefavored as candidates for key frames, so that the key frame is notblurred by camera motion. The set of key frames are ranked, according tofeatures such as camera fixation. Camera fixation indicates thepercentage of the entire video capture for which the camera was fixatedon a given region of the scene. Video key frames corresponding toregions of high camera fixation are ranked highly, as they belong toparts of the video of potentially high interest. The video summary canbe formed by merging snippets of video, wherein each snippet correspondsto a collection of video frames surrounding a highly ranked key frame.Video summarization parameters include the desired length of thesummary, minimum length of any snippet of video contained in thesummary, and the total number of snippets contained in the summary. Thecollection analyzer 106 can produce proposals for video summarizationshaving a variety of total lengths, number of snippets, and minimumsnippet lengths.

The collection analysis 108 contains a set of possible playbackinstructions that indicate portions of the video to play as a summary.For example, if the video is 1 minute long, than an 18 second version ofthe video is to play from 0:02 to 0:10 and from 0:20 to 0:30. A 12second version of the video is to play from 0:14 to 0:26. It is assumedthat videos contain both audio and image information, and the playbackinstructions describe playback for both audio and video, alternativelythere are a separate set of instructions for playback of audio and videoinformation.

The collection analyzer 106 can also produce the collection analysis 108that indicates information relevant to each image and video. Examples ofsuch information include identities of individuals present in the imagesand videos, activities occurring in the captured scene, location of thecaptured scene, and other significant non-person objects identified inthe scene. Identities of individuals present in image and videos can bedetermined by face detection and recognition algorithms, such asdescribed in U.S. Pat. No. 6,940,545 and in “Eigenfaces for Recognition”by Turk et al, Journal of Cognitive Neuroscience, 1991. Activitydetection in images and videos can be used to identify particularactivities of interest, such as weddings, parties, sporting events andperformances. The location of the captured scene can be estimated usingGPS data 329 collected at capture time. Other significant objects in thescene, such as pets, can be detected as in U.S. Patent ApplicationPublication 2010/0119155.

Information relevant to individual videos can be used as an additionalinput to the collection analyzer 106 for generating proposals for videosummarization. In particular, summaries can be biased to focus onspecific individuals or activities. Viewers 110 ₁ with strong socialconnections to the individuals or activities in the video can receive aversion of the summary that focuses on the specific individuals oractivities.

The collection analyzer 106 can also determine an overall quality of anindividual video using methods known in the art. One example of such amethod is described by Z. Wang in “Visual Quality Assessment Based onStructural Distortion Measurement”, Signal Processing: ImageCommunication, 2004.

A viewer 110 ₁ is also a member of the same social network as the user100. The social relationship between the viewer 110 ₁ and the user 100is used by the present invention to summarize the user's imagecollection 102 for presentation to the viewer 110 ₁. To this end, asocial distance analyzer 118 accesses the social network 116 todetermine a social distance 120 between the user 100 and the viewer 110₁. Determining the distance between two people in the social network 116can be performed by any of a number of methods.

In one simple version, the distance between the user 100 and the viewer110 ₁ is simply the shortest path in the graph G that represents thesocial network. For example, a friend of the user 100 has a distance of1, a friend of a friend has distance 2, and so on. In some cases, thesocial distance 120 is not simply a scalar, but also containscategorical information such as the types of relationships (e.g.familial relationship, friend, coworker, colleague, classmate) along theshortest path between the user 100 and the viewer 110 ₁. In anotherembodiment, the social distance 120 between the user 100 and the viewer110 ₁ is the shortest path among graphs having only one type of socialconnection (where the types of connections include familialrelationship, friend, coworker, colleague, classmate). For example, thesocial distance 120 can be “family 2” or “friend 1”. In anotherembodiment, the social distance 120 is found with the method describedin U.S. Pat. No. 7,788,260.

To illustrate a further alternative embodiment, the graph G is definedover all the connections in the network. For example, FIG. 1 contains 12individuals (A, B, C, D, E, F, G, H, I, J, K and M). The graph G isrepresented as a 12×12 matrix where nonzero entries indicate graphedges. A graph is a set of vertices (in our case, the individuals) and aset of pairs of vertices (in our case, the connections betweenindividuals or identified individuals in digital media assets 430). Thevalues of the nonzero entries indicate the weights of the graph edges(i.e. the strength of connections). The values in the matrix can becalculated using:

${G\left( {x,y} \right)} = {1{\prod\limits_{i \in B}^{\;}{d_{i}\left( {x,y} \right)}}}$

where d_(i)(x,y) is a value related to the presence of a connection. Theindicator i represents the type of friend connection (when i=1, theconnection is a friend connection, when i=2, the connection is anassociate connection, and when i=3 the connection is an imageconnection.) The set B is the set of indices such that d_(i)(x,y) isnonzero. Thus, the weight G(x,y) is the product of the weights of allthe existing connections between person x and person y. For example,d_(i)(x,y) can be set to 0.5 if individuals x and y have a direct friendconnection 424 between user accounts, and can be set to 0 otherwise.Note that in some embodiments, the weight for a direct connection variesbased on the connection type (e.g., relative, coworker, classmate,friend). Similarly, d₂(x,y) can be set to 0.75 if individuals x and yhave an associative connection 426 between a user account and a digitalmedia asset 430 and 0 otherwise, and d₃(x,y) can be set to 0.67 ifindividuals x and y have an image connection 428 within the digitalmedia asset 430. Note that other connections can also contribute toweights in the graph between individuals. For example, membership incommon or related groups, or similarity in interests between two peopleindicates some likelihood that these people will become or areacquainted.

The social distance between any two individuals 420 in the socialnetwork 116 is a function of the graph G. In the preferred embodiment,the social distance 120 s_(G)(i,j) is a function f( ) of featuresextracted from the graph G so that s_(G)(i,j)=f(G_(F)), where G_(F) arefeatured extracted from the graph. In one embodiment, the feature vectoris the length of the shortest path (the path with the smallest sum)between the people i and j. For example, in FIG. 1, the weights to otherpeople based on the shortest path from the owner of the person Econtaining the unidentified individual 432 to each of the others is:G_(F)=[0.5, 0.75, 0.5, 0.5, 0.0, 1.0, 1.0, 0.5, 1.0, 1.5, 1.0, 1.0]corresponding to the individuals 420 (A, B, C, D, E, F, G, H, I, J, K,M). This feature is based on the idea of finding the distance betweenindividuals 420 in the graph that represents the social network. Notethere are more than two weights possible for the candidate individuals.It can be seen that this approach has the desired affect that a smallerdistance is given to persons who are directly connected to the owner ofthe digital media asset 430 than to persons who are indirectly connectedto the owner of the digital media asset 430.

In other methods, the features are based on a set of paths between theuser 100 and the viewer 110 ₁. For example, considering only the directconnections between user accounts, there are 3 paths of length 2 (in thenon-weighted version of the graph G) between B and E (B-A-E, B-C-E,B-D-E) but only one path of length 2 between individuals E and G(G-H-E). Therefore, there is a stronger connection between B and E thanbetween E and G, and consequently the social distance 120 between B andE is smaller than between E and G.

Further, the graph can be thought of as representing a flow network andthe feature can be the amount of flow passing through a given node(i.e., individual in the network) when the flow source is theunidentified individual. Algorithms to analyze the flow in a network arewell known in the art, such as min-cut maxflow algorithms. Stillfurther, the graph can be though of as a resistor network in electricalcircuit design, with each edge in the graph having a certain resistance.Then, the social distance 120 between any two people is the resultingequivalent resistance between their corresponding nodes in the graph. Insome embodiments, other factors are considered when determining thesocial distance 120 between two people in the social network 116,including the frequency of interaction between the pair (e.g. the numberof messages exchanged in the past year), or the number or percentage ofimages or videos shared from one to the other of the pair that theviewer actually clicked on, viewed, commented on, or watched.

The social distance 120 between the user 100 and the viewer 110 ₁ ispassed to a parameter selector 122 for selecting social summarizationparameters 124 that determine how to summarize the user's imagecollection 102 for presentation to the viewer 110 ₁. The socialsummarization parameters 124 can come in a number of forms. For example,the social summarization parameters 124 can indicate that the viewer 110₁ should be able to access the entire image collection 102. This mightbe appropriate if the user 100 and viewer 110 ₁ are especially close,for example, daughter and mother, respectively. Alternatively, thesocial summarization parameters 124 can indicate that only N of theimages from the image collection 102 should be shared with the viewer110 ₁, and the total length of all videos should be not greater than Tseconds. For example, if the user 100 and viewer 110 ₁ are coworkers,then the social summarization parameters 124 indicate that no more than3 images and the total length of all videos should not exceed 36seconds. In another embodiment, the social summarization parameters 124indicate the maximum percentage of the total images and videos from aparticular user 100 to that viewer 110 ₁ over a given time period (e.g.1 day). The parameterization is determined for a viewer 110 ₁ byconsidering all users 100 that have uploaded image collections 102. Forexample, suppose a viewer Tom 110 ₁ has three friends with uploadedimage collections and has social distance 120 to the friends as follows:[0.5 1.0 2.0] (meaning that Tom is socially closest to the first friend,then less close to the second, and least close to the third). Tom has alimit of viewing 42 images per day, and the three friends have eachuploaded image collections with 150 images. In this case, the socialsummarization parameters 124 indicate that Tom will be presented 24images from the first friend, 12 from the second, and 6 from the third.Each user 100 is permitted to have:

$N_{i} = {T\frac{D_{i}^{- 1}}{\sum\limits_{n}^{\;}D_{n}^{- 1}}}$

Where T is the total number of images (or length of video) that theviewer 110 ₁ is interested in viewing over a certain time interval,D_(i) is the social distance 120 between the viewer 110 ₁ and aparticular user 100, and N_(i) is the maximum number of images (orlength of video) that can be contributed by the i^(th) user 100. Whenthe user has fewer images or total video length smaller than the limit,then the excess is passed on to the other users in the social network.For example, if the first user has an image collection of only 20 imageswhen the value of N for that user is 24, then the next user can exceedtheir value of N by 4.

The social summarization parameters 124 can also indicate that a certainviewer 110 ₁ can only see images or videos of a certain content. Forexample, a user 100 can decide that images containing his family canonly be viewed by social connections 424 classified as “relative” or“friend”, while images of the research project he is working on can onlybe viewed by social connections 424 classified as “co-workers”. Inanother example, a user 100 can belong to a specify organization such asa sailing club, and choose only to share images and videos containingsailing content with social connections 424 who belong to the same club.Similarly, the social summarization parameters 124 can indicate that acertain viewer 110 ₁ prefers images and videos having certain content.

The social summarization parameters 124 can be determined automaticallyby the parameter selector 122. Alternatively, the user 100 can designatesummarization parameters 124 for members of his or her social network.The viewer 110 ₁ can also designate summarization parameters 124 withrespect to a particular user 100. For example, a particular viewer 110 ₁can choose to view only one minute of video per day from a particularuser 100, and set summarization parameters 124 accordingly.

The collection summarizer 126 summarizes the images and videos 32 on theimage collection 102 in accordance with the social summarizationparameters 124 and produces a summarized image collection 128. Thisprocess is repeated for each viewer 110 ₂, 110 ₃, in the social network116 having a social distance 120 to the user 100 that meets a specificcriteria (e.g. there exists a direct connection between the viewer andthe user 100, or the viewer 110 ₁ has a social distance 120 less than 2from the user 100).

For summarizing a collection of images, a collection summarizer 126performs any of a number of methods known in the art. For example, onone preferred method, the collection summarizer simply takes the top Nimages of the image collection 102 based on an ordering of the images interms of image quality from the collection analysis 108. In otherembodiments, the collection summarizer 126 selects the N images of theimage collection that represent the collection, using for examplemeasures of mutual information between the selected images and the otherimages in the image collection 102. Such a method is described inModeling and Recognition of Landmark Image Collections Using IconicScene Graphs. X. Li, C. Wu, C. Zach, S. Lazebnik and J.-M. Frahm.Proceedings of the European Conference on Computer Vision, 2008.

In other embodiments, the collection summarizer 126 considers theidentity of and attributes of the viewer 110 ₁ in relation to thecontent of the image collection 102. For example, if the viewer 110 ₁ isa relative of the user 100, images in the image collection 102 that areidentified as containing family members can be ranked relatively higherthan images not containing family members. In another example, theviewer's 110 ₁ social connections can be considered, such that images inthe image collection 102 that are identified as containing friends ofthe viewer 110 ₁ can be ranked more highly than images not containingfriends of the viewer 110 ₁. In another example, the known interests ofthe viewer 110 ₁ can be considered, such that images in the imagecollection 102 that pertain to the viewer's 110 ₁ known interests (e.g.images of mountains, images of bicycling, images from Paris, images ofbaseball games) can be ranked more highly than images not pertaining tothe viewer's 110 ₁ known interests.

A viewer's 110 ₁ interests can be learned or known in many differentways. Such methods include analysis of the viewer's 110 ₁ own imagecollection, consideration of social clubs and groups of which the viewer110 ₁ is a member, and explicit tags listed by either the user or theviewer 110 ₁ that indicate interests of the viewer 110 ₁. For example, auser Caitlyn can have a friend Will who is known to be an avidoutdoorsman. Caitlyn can explicitly set a tag that indicates Will'sinterest in the outdoors such that the collection summarizer 126 favorsoutdoor pictures when generating a summarized image collection 128 forWill.

For summarizing a collection of videos, the collection summarizer 126performs any of a number of known methods in the art. Given a totalvideo time allocation of T seconds, the collection summarizer 126 caninclude whole videos according to a quality ranking provided by thecollection analyzer 108 until the time allotment is reached.Alternatively, the collection summarizer 126 can distribute the Tseconds of allotted video among the entire collection of videosaccording to their relative quality. For example, if Q_(i) is thequality of video i, it receives T_(i) of video time allocation accordingto the following formula:

$T_{i} = {\frac{Q_{i}}{\sum\limits_{i}^{\;}\; Q_{i}}{T.}}$

In order to prevent the dilution of the allocated time among too manyvideos, the time allocation can be restricted to the top N rankingvideos. In the above formula, if a given video is allotted more timethan the length of the video, the excess time can be distributed amongthe remaining videos. The above formula does not explicitly account forthe varying original lengths of different videos. The original length ofvideos can be included in the overall quality value assigned to eachvideo, such that longer videos receive longer time allocation thanshorter videos.

In another method of summarizing a collection of videos, the collectionsummarizer 126 weights the allotment of time among videos in thecollection according to the content of the videos relative to theinterests of the viewer 110 ₁. For a viewer 110 ₁ having known socialconnections to particular individuals, extra weight and hence greaterallocation of time can be assigned to videos containing thoseindividuals. Similarly, videos containing content of interest—particularplaces, or activities—can receiver greater allocation of time.

When the user 100 has a collection of multiple videos that aresummarized for the viewer 110 ₁, the summarized video can take the formof a collection of independent, smaller video summaries. Alternatively,the individual video summaries can be merged into a singlerepresentative video. This permits easy viewing of the video summaries,as only one video file needs to be accessed. Appropriate transitions,such as fades from one video to another, can be used to gracefullytransition from one video summary to the next.

After viewing the summarized image collection 128, the viewer 110 ₁ candecide to request additional content from the user 100. The request canbe for the viewer 110 ₁ to receive all of the remaining content from theuser 100. Alternatively, the viewer 110 ₁ can request limited additionalcontent, in which case the collection summarizer 126 can determine theadditional content to be shared. The viewer 110 ₁ can also requestspecific additional content from the user 100, such as any additionalimages containing a particular individual. The user 100 can choose toaccept or deny any request for additional content.

FIG. 4 shows illustrative results of the present invention for a socialnetwork containing four people Anne, Beth, Chris, and Dan. Both Anne andBeth post image collections to the social network. After finding thesocial distance 120 between Chris and Anne, and Chris and Beth, thendetermining social summarization parameters 124 for each, Chris receivesaccess to two images+a summarized video (36 seconds) from Anne, and twoimages from Beth. Meanwhile, Dan is socially closer (i.e. the socialdistance 120 between Dan and Anne is smaller than that between Chris andAnne) to Anne than Chris is, and consequently Dan receives five imagesand two summarized videos (one 1 minute 13 seconds, and one 23 seconds)from Anne Further, Dan is socially further from Beth than Chris is fromBeth, so Dan receives only one image from Beth. Note also that the orderof presentation of the summarized image collections is related to therelative social distances between the respective viewers and users (e.g.for Chris, Beth's images are ranked higher than Anne's, but to Dan, thereverse is true.)

The invention has been described in detail with particular reference tocertain preferred embodiments thereof, but it will be understood thatvariations and modifications can be effected within the spirit and scopeof the invention.

PARTS LIST

-   1 distance-   2 distance-   12 digital processor-   30 image capture device-   32 image or video-   43 compass-   45 tilt sensor-   47 accelerometer-   49 light source-   60 user controls-   62 user preferences-   69 enhanced image-   70 image processor-   90 display-   100 user-   102 image collection-   104 social network media summarizer-   106 collection analyzer-   108 collection analysis-   110 ₁ viewers-   110 ₂ viewers-   110 ₃ viewers-   116 social network-   118 social distance analyzer-   120 social distance-   122 parameter selector-   124 social summarization parameters-   126 collection summarizer-   128 summarized image collection-   322 RAM buffer memory-   324 real time clock-   328 firmware-   329 GPS sensor-   340 audio codec-   341 general control computer-   342 microphone-   344 speaker-   350 wireless modem-   358 mobile phone network-   370 internet-   375 image player-   420A individual-   420B individual-   420C individual-   420D individual-   420E individual-   420F individual-   420G individual-   420H individual-   420I individual-   420J individual-   420K individual-   420M individual-   422 persons-   422A person-   422D person-   422E person-   422F person-   422I person-   422M person-   424 connection-   426 associative connection-   428 image connection-   430 digital media assets-   431 video digital asset-   432 unidentified individual-   434 unidentified individual-   436 unidentified individual-   438 unidentified individual

1. A method for reducing the number of images or the length of a videofrom a digital image collection using a social network, comprising: (a)receiving a digital image collection captured by a user to be viewed bya viewer, wherein the viewer and the user are members of the same socialnetwork; (b) using a processor to access the social network to determinea relationship between a viewer and the user; (c) using the processor todetermine a set of summarization parameters based on the relationshipbetween the viewer and the user; and (d) using the processor to reducethe number of images or the length of the video from the digital imagecollection using the determined set of summarization parameters to beviewed by the viewer.
 2. The method of claim 1 further includingdisplaying the summarized digital image collection to the viewer.
 3. Themethod of claim 1, wherein when the digital image collection contains avideo, the summarization parameters include a parameter that limits thelength of the summarized video.
 4. The method of claim 1, wherein thedigital image collection contains a set of images, the summarizationparameters include a parameter that limits the total number of images inthe summarized image collection.
 5. The method of claim 1, wherein thesocial network is represented as a graph having vertices that representpeople in the social network and edges that represent the connectionsbetween people, and determining the relationship between a viewer andthe user includes determining a distance between the viewer and the userin the graph.
 6. A method for selectively reducing the number of imagesor the length of a video form a digital image collection using a socialnetwork individually for at least two viewers, comprising: (a) receivinga digital image collection captured by a user to be viewed by at leasttwo viewers, wherein the viewers and the user are members of the samesocial network; (b) using a processor to access the social network todetermine a relationship between each viewer and the user; (c) using theprocessor to determine a set of summarization parameters for each viewerbased on the relationship between each viewer and the user; and (d)using the processor to reduce the number of images or the length of thevideo from the digital image collection for each viewer using thedetermined set of summarization parameters to be viewed by each viewer.7. The method of claim 6 further including displaying the summarizeddigital image collection to each viewer.
 8. The method of claim 6,wherein when the digital image collection contains a video, thesummarization parameters include a parameter that limits the length ofthe summarized video.
 9. The method of claim 6, wherein the digitalimage collection contains a set of images, the summarization parametersinclude a parameter that limits the total number of images in thesummarized image collection.